Method and system for creating, administering and automating scoring of dimensional modeling constructed response items

ABSTRACT

The methodology of the present invention is implemented by a system which defines concepts about which a respondent is required to demonstrate knowledge of a relationship. The concepts are represented by respondent-manipulable devices that are uniquely associated with the concepts, and the respondent is able to manipulate attributes of the devices to create an arrangement of the devices representative of the respondent&#39;s understanding of the relationship of the concepts with which the devices are associated. The attributes of the manipulated devices are recorded and converted into a response pattern based on the cognitive relationships of the devices as manipulated by the respondent. One or more predefined rubric patterns are associated with the constructed response item and define possible cognitive patterns of the concepts as represented by the devices. The response pattern is compared to the rubric pattern to identify matches between the response pattern and the rubric pattern. Implications, or outcomes, are associated with each rubric pattern defined, and a scoring result is created based on the implications of the rubric patterns that match the response patterns.

This application is a continuation of Ser. No. 10/643,889 filed Aug. 20,2003, which in turn claims the benefit of U.S. Provisional ApplicationNo. 60/404,393 filed Aug. 20, 2002. The disclosures of the priorapplications are hereby incorporated by reference herein.

BACKGROUND

1. Field of the Invention

The present invention provides systems and methods that make teachingand assessment of students' understanding of relationships betweenconcepts and facts more interesting to students and teachers and alsodecreases the costs and elapsed time associated with creation,administration, and scoring of constructed responses for students.

2. Discussion of Related Art

Multiple choice type response items (i.e., questions) completely bindstudent responses to the question to the answer choices presented. Thistype of item must be constructed very carefully to avoid the problem of“smart test takers” guessing the answer to the problem based on theconstruction of the distracters (i.e., possible incorrect responses)versus the correct response. The binding of the item provides additionalinformation about the intent of the question when the student evaluatesthe complete set of possible choices. This means that the time requiredfor ensuring that the stimulus (i.e., a passage or graphic display aboutwhich questions are asked) was completely understood is reduced. Anymisunderstandings that are possible from the stimulus but which are notoffered as possible responses to the item can be discounted by thestudent.

Constructed response items, on the other hand, call for responses madeup in the mind of the student, rather than chosen from a list ofoptions. Paper and pencil, open, and computer or paper based textresponse constructed response items leave the students entirelyunbounded in their response. This openness nearly eliminates the errorin assessments based on “guessing”, but doesn't provide for any supportfor the student to ensure that the item's problem domain was clearlyunderstood.

Computer software applications are available which teach the participantabout relationships between various concepts. For example, children'sinteractive tutorial programs teach children about the relationshipsbetween different concepts by manipulating graphic icons on the computerscreen in accordance with instructions provided by the program. Forexample, the child may be requested to match one object (e.g., a key)with a letter associated with it with a complimentary object (e.g., akeyhole) with the same letter on it. The child may be asked to associatea baby animal (e.g., a puppy, a kitten, a calf, a piglet) with an adultanimal (e.g., dog, cat, cow, pig) of the same species based on the soundmade by or the appearance of the baby animal—for example, by clicking onthe baby animal and dragging it to one of a group of different adultanimals. The child may be asked to drag objects of different shapes intocorrespondingly-shaped “holes.” The child may be presented with part ofan incomplete pattern and a variety of objects from which to choose tocomplete the pattern by selecting the correct objects and placing themin the proper relationship with respect to each other.

Such exercises are more akin to restricted response assessment items;the participant is typically offered a limited choice of optionsavailable for manipulating the graphic icons. Moreover, such exercisesare primarily a teaching tool, rather than an assessment tool.Typically, the program will not allow the participant to make anincorrect choice—e.g., the child will not be allowed to place a circularobject into a triangular hole—, and if an incorrect choice is permitted,the participant is immediately advised in some manner that the choice isincorrect and encouraged to try another choice. In this regard, theprogram is teaching the participant about the correct relationships ofthe different concepts—e.g., the puppy goes with the dog, the kittengoes with the cat, the calf goes with the cow, and the piglet goes withthe pig—; it is not testing the extent of the participant's cognitiveunderstanding of the relationships of the different concepts.

SUMMARY OF INVENTION

Aspects of the invention provide for defining rubrics (i.e., a scoringtool, or a set of criteria, used to evaluate a student's response) interms of cognitive relationships between facts and/or concepts availablefor student manipulation in a constructed response in a useful,efficient, and cost effective manner. Aspects of the invention alsoprovide for defining a multiplicity of implications (i.e., outcomescorresponding to the student's response) based upon evaluation ofdifferent fact and/or concept patterns created by the student respondingto the constructed response items. Aspects also provide for design ofitems (i.e., questions or problems) which can be administered acrossmultiple administration modalities, including manual and computer-basedadministration. Aspects of the invention also provide for automatedscoring of all these administration modalities.

A system embodiment of the invention comprises communicating coupleddevices, including one or more content creation engines coupled to oneor more rubric engines coupled to one or more administrationengines—which may be either manual, computer-based, or both—coupled toone or more scoring engines. This communication can be within the samecomputer, over any communication network, public or private, including alocal area network (LAN), wide area network (WAN), telephone network,cellular phone network, pager network, and Internet, etc., or by anycombination of some or all of these communication pathways.

A content creation engine comprises a concept creator which enables anitem designer to create concepts or facts, a template selector whichenables an item designer to choose templates to be used for presentingthe content, and a representation linker which enables an item designerto define associations between concepts and one or more physical orvirtual devices, such as, blocks, balls, toys, images, sounds, videostream, text, etc. for representing concepts and which can bemanipulated by a person, with or without a computer system, to create apattern or arrangement of these devices as a response to an item.

A computer based administration engine comprises a representation enginewhich presents the devices which have been chosen to represent theconcepts to the user for manipulation, a manipulation engine to enablethe person responding to an item to manipulate the devices to create apattern, or arrangement of devices, and an attribute acquisition engineto retrieve and record the attributes of the devices (i.e., theposition, orientation, relation with respect to other devices, etc.)once the manipulation by the user is completed.

A manual based administration engine comprises a setup process describerengine to provide to the administrative user descriptions of the setupprocess and devices to be used by the student user, an interaction andmanipulation process engine to provide information on the processmanipulation of the devices, an attribute acquisition process engine forretrieving and recording attributes of manipulated devices, which mayinclude manual or computer automated aspects or both, and an attributetransformation engine to convert the attributes into the same format asthe output from the attribute acquisition engine of the computer basedadministration engine.

A rubric engine comprises a pattern definer for creating sets of conceptand/or fact relationships from predefined cognitive relationships and animplications definer for indicating a given pattern set's implications.That is, the pattern definer defines relationships (i.e., patterns) ofeach fact or concept associated with a constructed response item toother facts or concepts associated with the item. The defined patternswill be used to assess and evaluate the patterns created by the student.The present invention relies on comparisons of relationships betweenfacts or concepts—rather than comparisons of the absolute values ofattributes of the devices chosen to represent particular facts orconcepts—to assess and evaluate constructed responses. In this manner,the pattern(s) defined for assessing and evaluating a response for aparticular constructed response item will always be valid for that item,regardless of what devices are selected to represent the facts orconcepts involved and regardless of the modality chosen for presentingthe item.

An “implication” is information about processes or states for a systemor person as the result of a pattern being found in the student responseto an item. The invention allows for one or more implications to bedefined for each pattern. These implications can be used to define suchthings as the raw score for an item, the identification number orreference to the next item to be given to a student, a text string to bereturned to the student or printed in a report, the probability orlikelihood of the student knowing a particular standard, a sound to playduring administration, a graphic to be displayed on screen, possibly ina report or during an administration, an animation path definition toshow the student, etc.

A scoring engine comprises a pattern recognition and comparison engineto identify patterns, which exist both in the device attributes receivedfrom the administration engines and those from the rubric engine, animplications selector to allow for a subset of the available implicationtypes to be returned, and a results engine to return results in thedesired format.

Utilization of the invention leads to a greater flexibility inascertaining the level of understanding that a student has in thesubject area being assessed or instructed. There are significantadvantages to the use of this form of dimensional modeling item. Theadvantages of this invention include new types of response bounding,concept definition, rubric definition, administration flexibility, andautomated scoring speed and feasibility.

Constructed response items created using dimensional modeling accordingto the present invention provide a boundary of the item's problem domainmuch like that of multiple choice response items. The students can'tjust answer in anyway they choose, instead they must use the devicesprovided and may change only those attributes of the devices that theitem design allows to be changed. Errors in analysis of studentresponses due to students guessing at answers are reduced as compared tomultiple-choice response items because the number of possiblepermutations in the answer to the problem is generally higher than thenumber of possible choices practical with multiple-choice items, therebymaking pure guessing a less feasible option for the student. The type ofassessment item developed in accordance with the invention should alsoresult in lower instances of “clueing,” whereby the distracters and itemconstruction give clues as to the correct response to the item. Clueingis a problem common with multiple-choice type items.

Item creation using an embodiment of the invention involves creating aset of concepts by either creating new concepts or selecting conceptsalready created and stored and cataloged in a database, choosingadministration templates, selecting devices to represent the conceptswithin each template, defining a rubric consisting of patterns ofrelationships between concepts (i.e., conceptual relationships), asrepresented by the devices, and defining implications of the existenceof those patterns in the response. This method of creating an item canallow for the construction of items which have multiple parts, where theparts are dependant items, each of which can be administered eithermanually or using a computer, dependant on the needs of the studentand/or purpose of the test.

By way of example, item designers may use the invention to associateconcepts such as “Sun”, “Moon”, “Frog”, “Romance”, “Like”, “Dislike”,“Days”, “the number 3”, “the sound of a dog barking”, or “the letter A”,with uniquely-designated devices that can be manipulated by the user,such as, GIF images, AU sound files, text strings, toy balls, rulers,video tape, DVD, cassette tape, etc. During item administration, thestudent can demonstrate his knowledge of the conceptual relationshipsthat exist between these concepts by manipulating these devices inrelationship to each other.

The rubric for each item is a set of relationships between theseconcepts and is independent of the devices associated with each concept.This means that the student could use different administration processesto demonstrate the knowledge they have about the concepts assessed inthe item once the concepts have been generated and associated withspecific device representations. By utilizing a conceptual relationshipset definition for correct and incorrect possible resulting responsesfrom an item, the item designer is able to specify in a very compact,intuitive way the implications of any or all of the resulting patternsthat are valuable for instruction or assessment.

This conceptual rubric is preferably written in a formalized languagethat will be easy for the item writers and editors to assimilate becauseit will build upon the manner in which constructed response rubrics arewritten using natural language today. The use of this formalized rubriclanguage also eliminates the need for the item writer to interactdirectly with a programming environment or understand the mathematicalconstruction of a solution set.

Since the rubrics and concepts defined for the items are not dependanton the actual devices associated with the concepts, the same itemconceptual definition can be used to allow the student to demonstrateknowledge of the conceptual relationships specified in the item designusing various technologies, or using manual administration processes.This allows the effort in creating the concept definitions andgenerating the types of relationships being assessed to be reused acrossmultiple testing systems. It also allows for a greater ease in use ofitems designed today for use with tomorrow's technological advances. Italso allows for students with various special needs to be assessed onthe same conceptual relationships as other students without such specialneeds by merely selecting devices that are appropriate to the student'sneeds and abilities as the devices associated with the specificconcepts.

The scoring engine uses the formal concept definitions from the rubric,making automated scoring feasible, reproducible, and fast.

The scoring engine receives device attributes from the administrationprocess and converts them to conceptual patterns, which it then comparesto patterns defined in the rubric. By passing device attributes to thescoring engine, rather than scores, multiple administrations can use thesame scoring logic.

Once the scoring engine identifies all of the pattern matches, anddetermines if there are any completed sets of concept patterns withinthe response which match those defined in the rubric, it then identifiesthe resulting set of implications corresponding to the matched conceptpatterns defined in the rubric and returns a set of implicationsfiltered by the type of implication desired in that particular testing.This allows for the same item design to be utilized in differentassessment scenarios, such as returning a raw score for a standardizednorm referenced test, a domain to test next for a diagnostic assessment,or other test purpose implication requirements.

Thus, the respondent will be presented with a number of devices, eachassociated with a different concept, and will be asked to demonstratehis or her knowledge of the conceptual relationships associated with aparticular scenario presented to the respondent in a stimulus. Therespondent will then manipulate one or more attributes (e.g., its color,size, orientation, location, sound, volume, texture, pattern, smell,taste, elasticity, weight, transparency, absorbency, reflectivity,shape, electrical charge, magnetism, temperature, conductivity,composition, intensity, perspective, emotion, etc.) of one or more ofthe devices to create an arrangement of the manipulated devices thatreflect the respondent's understanding of the conceptual relationshipassociated with the scenario.

The response will not be completely unbounded in that the arrangementmust be created using one or more of the devices presented, of whichonly certain attributes will be manipulable. On the other hand, withinthe constraints of the devices presented and the attributes which may bemanipulated, the response is otherwise unbounded. The arrangementcreated is converted to a conceptual pattern, and the conceptual patterncreated by the respondent is compared to conceptual patterns that aredefined by the scoring rubric, which include correct patterns (more thanone conceptual pattern can be correct), incorrect patterns, andpartially correct patterns. The extent to which the conceptual patterncreated is correct, incorrect, or partially correct is informativebecause it does more than tell the educator that the respondent does ordoes not understand certain concepts; it informs the educator as to whatthe respondent does actually understand. Such information can help theeducator predict what less advanced concepts the respondent understandsor needs to learn as well as what more advanced concepts the respondentmay already understand or is ready to learn.

In this regard, the present invention is different from the educationalcomputer applications described above. Responses are far more unboundedthan in the educational programs, and the respondent is allowed tocreate incorrect or partially correct arrangements, as well as correctrelationships. And such incorrect and partially correct arrangements arereviewed and analyzed to derive information from them regarding therespondent's level of understanding of the conceptual relationships.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates a block diagram of dimensional modeling systemaccording to an embodiment of the invention.

FIG. 2 illustrates a block diagram of a computer according to anembodiment of the invention.

FIG. 3 illustrates a block diagram of a content creation engineaccording to an embodiment of the invention.

FIG. 4 illustrates an item design for a dimensional modeling item in 3dimensions (x, y, t) for an item designed for assessing specificknowledge of the solar system according to an embodiment of theinvention.

FIG. 5 illustrates a template definition according to an embodiment ofthe invention.

FIG. 6 illustrates dimensional modeling in 2 dimensions (x, y) forcomputer-based administration for an item designed around assessment forsolar system content according to an embodiment of the invention.

FIG. 7 a illustrates dimensional modeling with height variance forcomputer-based administration for an item designed around assessment forsolar system content according to an embodiment of the invention.

FIG. 7 b illustrates dimensional modeling with color variance forcomputer-based administration for an item designed around assessment forsolar system content according to an embodiment of the invention.

FIG. 7 c illustrates dimensional modeling with orientation variance forcomputer-based administration for an item designed around assessment forsolar system content according to an embodiment of the invention.

FIG. 7 d illustrates dimensional modeling in 1 dimension (t) forcomputer-based administration for an item designed around assessment forsolar system content according to an embodiment of the invention.

FIG. 8 illustrates a possible resulting pattern for the 2 dimensionalmodeling of FIG. 6 according to an embodiment of the invention.

FIG. 9 illustrates another possible resulting pattern for the 2dimensional modeling of FIG. 6 according to an embodiment of theinvention.

FIG. 10 is a table illustrating the correspondence between concepts andspecific devices according to an embodiment of the invention.

FIG. 11 is a flowchart illustrating a content creation process accordingto an embodiment of the invention.

FIG. 12 illustrates a block diagram of a rubric creation engineaccording to an embodiment of the invention.

FIG. 13 is a table illustrating example concept relationships andexample mathematical definitions for various administration typesaccording to an embodiment of the invention.

FIG. 14 is a flowchart illustrating a pattern definition processaccording to an embodiment of the invention.

FIG. 15 illustrates a scoring rubric for the illustration of FIG. 6according to an embodiment of the invention.

FIG. 16 is a flowchart illustrating an implications definition processaccording to an embodiment of the invention.

FIG. 17 illustrates a manual administration engine according to anembodiment of the invention.

FIG. 18 is a flowchart illustrating a manual administration processaccording to an embodiment of the invention.

FIGS. 19 a-19 b illustrate a manual administration for the item designillustrated in FIG. 4 according to an embodiment of the invention.

FIG. 20 illustrates a digital recording of a manual administrationaccording to an embodiment of the invention.

FIGS. 21 a-21 d illustrate dimensional data acquisition and attributeacquisition for a manual administration of the item illustrated in FIG.19 a-19 b according to an embodiment of the invention.

FIG. 22 illustrates a computer-based administration engine according toan embodiment of the invention.

FIG. 23 is a flowchart illustrating a computer-based administrationprocess according to an embodiment of the invention.

FIG. 24 illustrates a computer-based administration for the item designillustrated in FIG. 4 according to an embodiment of the invention.

FIG. 25 illustrates dimensional data acquisition and attributeacquisition for a computer-based administration of the item illustratedin FIG. 24 according to an embodiment of the invention.

FIG. 26 illustrates a block diagram of a scoring engine according to anembodiment of the invention.

FIG. 27 is a flowchart illustrating an automated scoring processaccording to an embodiment of the invention.

DETAILED DESCRIPTION

Turning to FIG. 1, a suitable dimensional modeling system example 1500is illustrated according to an embodiment of the invention. As shown,dimensional modeling system 1500 comprises one or more (up to N) contentcreation engines 1700, 1700N (content creation means) for creatingconcepts, linking (or associating) those concepts to representationaldevices, and selecting templates for administration types. Dimensionalmodeling system 1500 also comprises one or more (up to N) rubric engines2000, 2000N for defining conceptual relationship patterns and anyimplications of the patterns. Dimensional modeling system 1500 alsocomprises at least one administration engine (administration means),which may be one or more (up to N) manual administration engines 1800,1800N and/or one or more (up to N) computer-based administration engines1900, 1900N. The administration engines allow the user to manipulate thevariable attributes of the devices representing concepts created in thecontent creation engine 1700 and return the resulting attributes toscoring engines. Dimensional modeling system 1500 also comprises one ormore (up to N) scoring engines 2100, 2100N (scoring means) for patternrecognition and comparison of responses from administration engines1800, 1800N, 1900, and/or 1900N with the pattern(s) defined by therubric engine(s) 2000, 2000N (rubric means), and for defining aresulting set of implications and returning the desired implicationtypes. Dimensional modeling system 1500 utilizes communication system1511 to communicate between its elements. Communication system 1511 canconsist of public or private networks, as, described above, or mayutilize mobile media such as floppy disk, CD-ROM, computer-readableprinted media such as bar codes, smart cards, magnetic stripe cards,natural voice communication, manuscript correspondence, etc.

In FIG. 2, computer 1600 may be used by any of the engines ofdimensional modeling system 1500 for all or some of its elements, withthe exception of manual administration engine(s) 1800, 1800N which mustinclude a manual administration process, along with its other methodsthat may or may not utilize computer 1600.

Computer system 1600 comprises elements coupled via communicationchannels (e.g., bus 1611) including one or more special purposeprocessors 1601, such as a Pentium®, or PowerPC®, digital signalprocessor (“DSP”), etc. System 1600 elements also include one or moreinput devices 1603 (such as a mouse, keyboard, joystick, microphone,remote control unit, camera, tactile, biometric, or other sensors,etc.), and one or more output devices 1605 (such as a suitable display,joystick feedback components, speakers, actuators, etc.), in accordancewith a particular application.

System 1600 also includes a computer readable storage media reader 1607coupled to a computer readable storage medium 1609, such as astorage/memory device or hard or removable storage/memory media; suchdevices or media are further indicated separately as storage device 1615and working memory 1617, which can handle hard disk variants,floppy/compact disk variants, digital versatile disk (“DVD)” variants,smart cards, read only memory, random access memory, cache memory, etc.,in accordance with a particular application. One or more suitablecommunications interfaces 1613 can also be included, such as, a modem,DSL, infrared transceiver, etc., for providing inter-devicecommunication directly or via one or more suitable private or publicnetworks, such as those already discussed.

Working memory 1617 further includes operating system (“OS”) 1619elements and other programs 1621, such as, application programs, mobilecode, data, etc., for implementing system 1500 elements that might bestored or loaded therein during use. The particular OS can vary inaccordance with a particular computing device, features, or otheraspects in accordance with a particular application (e.g., Windows, Mac,Linux, Unix, or Palm OS variants, a proprietary OS, etc.). I/O orenvironmental alternatives capable of being utilized with variousoperating systems can also be utilized, including, but not limited to,graphical user interfacing, pen-based computing multimedia, handwriting,speech recognition/synthesis, virtual/augmented reality, or 3-Daudio/visual elements.

Various programming languages or other tools can also be utilized, suchas, for example, Java, Python, and HTML. The above noted interfaces maybe written in Macromedia Flash™ using drag-and-drop and vector graphicsadvanced capabilities to manipulate device attributes. HTTPcommunication can be used to deliver resulting device attributes to thescoring engine. One or more elements of system 1600 can, however, beimplemented in hardware, software, or a suitable combination. Whenimplemented in software (e.g., as an application program, object,downloadable, servlet, etc.), in whole or in part, an element of system1600 can be communicated transitionally or more persistently from localor remote storage to memory for execution, or another suitable mechanismcan be utilized, and elements can be implemented in compiled orinterpretive form. Input, intermediate, or resulting data or functionalelements can further reside transitionally or more persistently in astorage medium, cache, or more persistent volatile or non-volatilememory (e.g., storage medium 1609, or memory 1617) in accordance with aparticular application.

FIG. 3 illustrates a content creation engine 1700 according to anembodiment of the invention. Content creation engine 1700 comprises oneor more concept creators 1701, which allow an item designer to select apreviously created and stored concept or create a new concept and givethe new concept a description. The concept descriptions can include textstrings, mathematical formulae, sound files, video streams, image files,numbers, or any other data type including text and binary files.Concepts can be re-used across multiple items (concepts include suchthings as the planet Earth, Love, mathematical addition, phonemes, atheme, a main character, bouncing, potential energy, molecular valence,narrow-mindedness, etc.).

FIG. 4 illustrates an example item design according to an embodiment ofthe invention. In the example shown in FIG. 4, the item design isidentified by a topic, e.g., “Phases of the Moon,” and a uniqueidentifying attribute, such as an ID number. An item name, e.g., “Phasesof the Moon Calendar Model”, can provide an even more specificidentification of the item. The item shown in FIG. 4 requires studentsto demonstrate knowledge of the relative positions and orientations ofthe concepts associated with this item—namely, the sun, the moon, andthe earth—for different lunar phases. More specifically, the student isasked to show the relative positions of the earth, the sun, and the moonfor a particular phase of the moon corresponding to a particularcalendar date. Variable attributes, in this case, the two-dimensionalspatial and temporal coordinates (x, y, t), of each of the concepts aredefined. These attributes can be manipulated by the respondent, andtheir final values will determine the relations of the concepts (i.e.,the conceptual pattern), thereby allowing an assessment of thecorrectness of the test-taker's response.

Although shown in FIG. 4, it is not necessary to define the variableattributes in the item design; they may be defined through templateselection.

Content creation engine 1700 also includes template selector 1703, whichallows the item designer to select from a pool of templates for computeror manual administration, or both. These templates define the devicetypes and attributes which are variable for the devices. Templatescontrol what types of devices and which attributes of those devices canbe modified. For example, one template might allow for image devices tobe used and their x, y spatial positions to be manipulated. Examples ofother device attributes that can be manipulated include, color and/orbrightness, sound and/or volume, texture, pattern, smell, taste,elasticity, weight, transparency, absorbency, reflectivity, shape,electrical charge, magnetism, temperature, conductivity, composition,intensity, perspective, emotion, etc.

An example of a template definition is shown in FIG. 5. The templatedefinition includes a unique identifying attribute, such as ID No.00345, and a name, e.g., “2-D Position Modeling Template.” The templateincludes a short description of the template formats and implementation,device types, and variable attributes. The example shown is a templatefor use in connection with a constructed response item requiring thetest taker to demonstrate knowledge of the relative positions andorientations of two or more concepts in a two-dimensional space.Finally, the template includes an output description and the output dataformat.

Template implementations can be expressed in a variety of computer andnatural languages. FIG. 6 shows a screen shot of an example of aMacromedia Flash™ implementation of an item 500 employing the templatedefinition shown in FIG. 5 which includes a menu 501 of devicesassociated with particular concepts, including the planet Earth(represented by device 503), the Moon (represented by device 507), theSun (represented by device 505), persons in different orientations(represented by devices 509), trees (represented by devices 511), landforms (represented by devices 513), clouds (represented by devices 515),the earth's surface (represented by device 517), and the earth as a flatbody (represented by device 519). The X and Y coordinates of each devicecan be manipulated by the student by clicking on the image or animationand dragging it to a new location.

Other Macromedia Flash™ template examples 520, 540, 560, and 900, whichillustrate items in which attributes other than X and Y coordinates maybe varied, are shown in FIGS. 7 a, 7 b, 7 c, and 7 d, respectively.

The dimensional modeling item 520 shown in FIG. 7 a is a simple bargraph in which the height of each of four bars may be adjusted by thestudent to demonstrate knowledge of the amounts of daylight in New YorkCity in each of the four seasons of the year.

The dimensional modeling item 540 shown in FIG. 7 b includes arepresentation of the sun 541 and the earth 543 located adjacent to thesun. The earth 543 is divided in two by longitudinal line 547 andincludes a number of triangular segments 545. The student candemonstrate an understanding of the relationship between daytime andnighttime and the relative orientations of the sun 541 and the earth 543by clicking on (or otherwise darkening) those of the segments 545 thatwould be dark (i.e., nighttime) in correspondence with the sun 541/earth543 orientations shown.

The dimensional modeling item 560 shown in FIG. 7 c includes arepresentation of the sun 569 and earth 567 on its orbit around the sun.Three mutually orthogonal axes 561, 563, and 565 extend from the earth567. The student may demonstrate an understanding of the rotation of theearth 567 by manipulating (e.g., clicking on and dragging) one of theaxes 561, 563, 565 to cause the earth to spin around that axis.

Finally, in the dimensional modeling item 900 shown in FIG. 7 d, a menu901 includes eight representations 905 of the moon, each a differentphase, and a time line 903. The student may demonstrate knowledge of themonthly progression of the lunar phases by placing the eightrepresentations 905 in the proper order on the time line 903.

FIGS. 8 and 9 show examples of possible responses to the item of FIG. 6.In FIG. 8, the items 505, 503 and 509 have been manipulated in responseto the stem “Drag some of the pictures on the left to make a picture ofyou standing on the planet Earth at NIGHT” to represent that at night aperson 509 is on the side of the earth 503 opposite the sun 505. In FIG.9, various items have been manipulated in response to the same stimulusto demonstrate that at night the moon 507 is generally above a person509 standing on the earth's surface 517 with trees 511 and other landforms 513 properly oriented on the earth's surface 517 and the sun 505below, (or beneath or behind) the earth's body 519. The arrangementsshown in FIGS. 8 and 9 are created from the same group of availabledevices, and, although the arrangements shown in the two figures aresubstantially different, both define a conceptual pattern indicatingthat the creator of each arrangement has a generally correct cognitiveunderstanding of the relationships between concepts such as you, themoon, the sun, and the earth.

As further shown in FIG. 3, concept creation engine 1700 also includerepresentation linker 1705 which allows the item designer to link, orassociate, concepts to manipulable devices available in the templates.Concepts previously created may already have these linkages. FIG. 10illustrates device linkage examples for concepts such as the sun, theearth, the moon, time, a full moon, a half waning moon, a new moon, anda half waxing moon.

FIG. 11 is a flowchart illustrating a content creation process 100according to an embodiment of the invention. In step 101 of the process100, the item designer selects one or more template(s) corresponding tothe type of devices and manipulable attributes that are appropriate forthe item(s) being designed. In step 103, the item designer defines oneor more concepts corresponding to the item(s) being designed.Preferably, the number of devices defined by the template is at least asgreat as the number of concepts. Finally, in step 105, each deviceselected as part of step 101 is linked to one of the concepts defined instep 103.

FIG. 12 illustrates a rubric engine 2000 according to an embodiment ofthe invention. The rubric engine 2000 allows the designer to create aset of conceptual rubric patterns, or device arrangements, using patterndefiner 2001, that define device arrangements that correspond to correctand incorrect responses that are possible with the templates selectedusing template selector 1703 of the content creation engine 1700 (seeFIG. 3). These pattern sets may define all possible patterns into whichthe available devices can be manipulated or only some subset of allpossible patterns. Each of the rubric patterns consists of sets ofsub-patterns consisting of relationships between concepts (asrepresented by representational devices) defined using concept creator1701 of the content creation engine 1700. These concept relationshipsuse a formal, defined language, an example of which is shown in FIG. 13in which relationship definitions between different concepts representedby <A>, <B>, and <C>, each having positions (x_(A), y_(A)), (x_(B),y_(B)), and (x_(c), y_(c)) or (x_(A), y_(A), z_(A)), (x_(B), y_(B),z_(B)), or (x_(c), y_(c), z_(c)) are shown.

FIG. 14 illustrates a pattern definition process 120 from an exampleembodiment of pattern definer 2001. The pattern definition process 120is applied for each pattern in the rubric, beginning at step 121. Asub-pattern is created (step 123) by selecting a concept that isrelevant to the subpattern (step 125) and defining the relationship ofthe selected concept to the 1-n other concepts (step 127) that have beendefined in the content creation process (see FIG. 11). Not all conceptsneed to be relevant to every pattern in a rubric. The relationshipsbetween concepts consist of one or more predefined cognitive, physicalrelationships between concepts, and include relationships, such as“above”, “opposite to”, “before”, “in front of”, etc. Whether there areadditional subpatterns to be created is determined at step 129. If thereare additional subpatterns to be created, the process returns to step123; if there are no additional subpatterns to be created, the processproceeds to steps 131 and 133. At step 133, it is determined whethereach concept is used in at least one pattern. If each concept has beenused in at least one pattern of the rubric, the process 120 is complete;if not, the process 120 returns to step 123 so that additionalsubpatterns can be created for the unused concepts, if desired.

Rubric engine 2000 also allows the designer to create a set ofimplications for each pattern using implications definer 2003.Implications, which consist of a type and value pair, define an outcomethat occurs if the conceptual pattern created by the student matches therubric pattern with which the particular implications are associated.Type and values can be defined using any data type, including any kindof text, binary, or analog data.

A partial example of a scoring rubric is illustrated in FIG. 15. FIG. 15shows implication types and values for different possible patterns andsubpatterns created by the student manipulating devices in response to astimulus requiring the student to demonstrate knowledge andunderstanding of the relative positions of the sun, the earth, and aperson at night time. Only the first pattern/subpattern is correct. Thepossible implication types include “score”, “reward”, “correct”,“knowledge”, and “navigation”. Implication values for “score” would bethe numerical score the student receives for the response. Implicationvalues for “reward” might be computer files that create a visual image,such as a gold star or happy face for a correct response or a sad facefor an incorrect response, and/or an audible signal, such as a bell or a“hurrah” for a correct response or an “oops” or a “good try” for anincorrect or partially correct response. Implication values for“correct” include a binary “1” for a correct response or a “0” for aresponse that is not correct. Implication values for “knowledge” includea description of the student's level of knowledge estimated from thestudent's response. Navigation implications provide an ID (i.e., anaddress) for the following question for use in an adaptive testingformat in which the next question to be presented is selected based onthe extent to which the student gets the present question correct.

The implications type is useful in allowing the same item to be utilizedfor multiple test purposes. For example, the “score” type could be usedfor norm reference tests, whereas the “reward” type might be used forassessment integrated with instruction.

FIG. 16 illustrates an implications definition process 140 from anexample embodiment of an implications definer 2003. Starting with step141, for each pattern defined for a scoring rubric, an implication iscreated (step 143) by selecting an implication type (step 145) andentering or selecting an implication value (step 147). Different typesof implications may have different structures for the implication value.For example, referring to the sample rubric shown in FIG. 15, if theimplication type is “raw score” the value may be “3”, for anotherexample if the implication type is “knowledge”, the value may be{(“earth goes around sun”, 0.90), (“lack of sun defines night”, 0.45)},and for another example, if the implication type is “reward”, the valuemay be the image file “goldstar.jpg”.

If there are more implications to be defined for the rubric pattern(step 149), the process 140 returns to step 143. If there are no moreimplications for the rubric pattern, the implications definition process140 is complete as to that process.

FIG. 17 illustrates a manual administration engine 1800 according to anembodiment of the invention. Manual administration engine 1800 includessetup process describer engine 1801 which can provide computer or manualbased instructions for setting up for the interaction and manipulationprocess 1803.

FIG. 18 illustrates a manual administration 300 from an exampleembodiment of a manual administration engine 1800. The process 300begins by providing physical devices to a user (step 301). The physicaldevices are objects or mechanisms that can be manipulated and includethings such as balls, blocks, toys, index cards, video or audiorecording tape, spinners, etc. The user then, in step 303, responds to aconstructed response stimulus, by modifying one or more attributes ofone or more of the physical devices provided in step 301. In step 305,the user's response to the constructed item, as represented by thephysical device(s) modified by the user, is recorded. At least twooptions for recording the response are contemplated: digital and manualrecording.

For digital recording the attribute(s) of the device(s) that weremodified by the user are recorded at step 307 by, for example, digitalphotographs, radio frequency positional locators, etc. In step 309, thesystem generates a composite list of modified and unmodified uservariable attributes, and in step 311, the list of attributes generatedin step 309 are digitally stored.

For manual recording, at step 313, the administrator can record eitherthe user-modified device attributes (step 315) or the resultingcognitive relations of the devices following modification by the user(step 317). If attributes are recorded, the process 300 proceeds to step309. If cognitive relationships are recorded, the relationships aredigitally stored and then converted to attribute values before theprocess 300 proceeds to step 309.

FIGS. 19 a and 19 b illustrate an example manual administration, whichshows images of the setup for administration of an item involving Earth,Sun, and Moon concepts and respective devices (i.e., variously sizedballs 1100, 1102, 1104). A document provided by setup process describerengine 1801 provides setup and procedure information to the examinationproctor. The proctor asks the student to position the balls to correctlyindicate the positions of the Sun, Moon and Earth on four different daysof the month (associated with calendar 1106) corresponding to the phasesof the Moon: Full, Half Waning, New, and Half Waxing. FIG. 19 a showsthe setup, and FIG. 19 b shows the positions of the devices at somepoint during the interaction and manipulation process 1803.

FIG. 20 illustrates a digital recording process and setup of manualadministration 330 in which three cameras 331, 333, 335 are used totriangulate the X, Y, Z positions of three balls 337, 339, 341 sittingon a surface 343. For 2-D triangulation, only one camera is necessary.FIGS. 21 a-21 d uses a single camera showing an example of one type ofmanual administration data acquisition of the setup and interaction andmanipulation of FIG. 19 a-19 b. Four photographs shown in FIGS. 21 a, 21b, 21 c, and 21 d show the relative X, Y positions of the ball devicesrepresenting the Earth 1100, Moon 1104, and Sun 1102 concepts.

Attribute acquisition process 1807 (see FIG. 17) may be either a manualattribute acquisition process 1809, or a computer automated attributeacquisition process 1811, or both. For this example, the teacher entersthe date order and uploads the photographs using a web interface.Acquisition of the photographs may be a manual process and input byscanner, or may be automated using “web cams”. For this example,acquisition of the X, Y position of the balls is accomplished using acomputer automated attribute acquisition process by processing theimages from the manual administration data acquisition resulting in theX, Y positions for the center of each of the balls represented by axes1209. For this example, an attribute transformation engine 1805 combinesmanual input from the order of the days of the month with the order ofthe images and the X, Y position of the balls to create position andtime X, Y, T information for each of the concepts to be sent to scoringengine 2100.

The components of the manual administration engine are interconnected bylink 1813, which may be an electronic communication link, a manualcommunication link, or a combination of the two.

FIG. 22 illustrates computer based administration engine 1900, accordingto an embodiment of the invention. Computer based administration engine1900 comprises a representation engine 1901, which may comprise a webpage, Macromedia Flash™ application, Java application, virtual realityenvironment, or any other software or firmware application or componentthat allows for digital manipulation of virtual devices that results incontrol over one or more computer components such as those described foroutput device 1605 above.

Computer based administration engine 1900 also comprises a manipulationengine 1903 which provides the student with access to devicesrepresenting concepts for manipulation of attributes of the devices.Manipulation engine 1903 may be a web page, Macromedia Flash™application, Java application, virtual reality environment or anothersoftware or firmware application or component that allows for digitalmanipulation of virtual devices that allows control by one or morecomputer components such as those described for input devices 1603.

Computer based administration engine 1900 also comprises attributeacquisition engine 1905 which acquires the attributes of devicespresented by representation engine 1901 after manipulation by thestudent using manipulation engine 1903. Attribute acquisition engine1905 may be a software or firmware component that may be implemented invarious programming languages such as Java, Python, Flash ActionScript,JavaScript, etc.

FIG. 23 illustrates computer based administration process 200, which isan example process of an embodiment of computer based administrationengine 1900. In step 201 virtual devices are provided to a user via, forexample, computer internet browser, a personal digital assistant, cellphone, telephone, or other electronic device. In step 203, the usermodifies one or more attributes of the one or more virtual devicesprovided in step 201. In step 205 all user variable attributes of thevirtual items are stored, whether or not the attributes were modified bythe user.

FIG. 24 shows a screen capture of a computer administration of the sameitem design manually administered in the example of FIGS. 19 a-19 b and21 a-21 d.

FIG. 25 shows the data acquisition for the computer based administrationof FIG. 24. The data acquisition includes a template, in this case “2-DPosition Modeling Template” and an item name, “Phases of the MoonCalendar Model.” Data acquisition for the example shown in FIG. 25includes the devices, sun, earth, and moon and their respectiveattributes, (x-position, y-position, date) for each of the four datesfor which the user is required to show the relative positions of theconceptual devices (i.e., the sun, the earth, and the moon)corresponding to four different lunar phases.

Scoring engine 2100 of FIG. 26 receives device attribute informationfrom the administration engines 1800-1800N and/or 1900-1900N of FIG. 1via communication system 1511 and converts the device attributes intoone or more response patterns defined, not by the values of theattributes, but by the relationships between devices. Such relationshipsmight include: next to, within, above, below, to the right (or left) of,longer (or shorter) than, larger (or smaller) than, darker (or lighter)than, etc. Scoring engine 2100 comprises a pattern recognition andcomparison engine 2101 which utilizes formal definitions ofrelationships as shown in FIG. 13 to recognize relationships in the datathat results from the administration engine and compare it to the rubricpattern(s) of this item created using rubric engine 2000-2000N. Patternrecognition and comparison engine 2101 returns a set of pattern matchesbetween the rubric and the response. Scoring engine 2100 also comprisesimplication selector 2103, which allows the selection of a subset of theimplications of the patterns returned by pattern recognition andcomparison engine 2101. Scoring engine 2100 also comprises resultsengine 2105 which formats the results in the format desired by thereceiving party.

FIG. 27 is a flowchart illustrating an automated scoring process 400according to an embodiment of the invention. The process 400 is carriedout for each pattern (step 401) and each sub-pattern (step 403) in thescoring rubric. For each sub-pattern, the response pattern is comparedto the rubric sub-pattern (step 405) to determine if the responseincludes the sub-pattern. If any sub-pattern of the scoring rubric isnot found in the response, the associated pattern is not found in theresponse, and the process 400 returns to step 401 to check the responsefor the next pattern of the scoring rubric. If the sub-pattern does notexist in the response and the pattern includes additional sub-patterns(as determined at step 407), process 400 returns to step 403 to checkthe next sub-pattern associated with the pattern.

If all subpatterns associated with a pattern of a scoring rubric existin the response, the associated pattern is recorded as existing in theresponse at step 409. Next, the desired implication types are selectedat step 411. Depending on the circumstances in which the constructedresponse item is used, it may be desirable to record for use only asubset of the available implication types. For example, it may bedesired to record only the “raw score” implication type or the“knowledge” implication type, rather than both. At step 413, theimplication type(s) selected for the pattern are recorded.

Although a preferred embodiment of the system and methodology of thepresent invention is specifically illustrated and described herein, itwill be appreciated that modifications and variations of the presentinvention are covered by the above teachings and within the purview ofthe appended claims without departing from the spirit and intended scopeof this invention.

1. A computer-readable storage media having stored thereoncomputer-executable instructions for causing a computer to perform,either alone or in conjunction with one or more users, a method foradministering a constructed response assessment item to a respondent,wherein the computer performs steps comprising: presenting a stimulusresponse stem requiring the respondent to demonstrate knowledge of arelationship between two or more concepts; providing the respondent atleast one device uniquely associated with each of the two or moreconcepts, at least one device having at least one attribute that can beselectively manipulated by the respondent to change a relationship ofthat device with respect to at least one other device to construct anarrangement of devices representing a relationship between the conceptswith which the devices in the arrangement are associated; storing theselectively manipulable attributes of the devices in the arrangementconstructed by the respondent; converting the stored attributes into aconceptual pattern associated with the arrangement constructed by therespondent; and comparing the conceptual pattern of the arrangementconstructed by the respondent with a conceptual pattern of a pre-definedscoring rubric to determine whether or not the conceptual patternassociated with the arrangement constructed by the respondent matchesthe conceptual pattern of the scoring rubric.
 2. The media of claim 1,wherein said pre-defined rubric includes a conceptual patterncorresponding to a correct relationship between the two or moreconcepts.
 3. The media of claim 1, wherein the devices comprise virtualitems displayed on the computer that the respondent can electronicallymanipulate and/or position with respect to each other to create avirtual arrangement of the devices demonstrating a relationship betweenthe concepts with which the devices are associated.
 4. The media ofclaim 1, wherein the attribute of each device that can be manipulatedincludes one or more of size, length, height, spatial, orientation,color, direction of movement, speed of movement, sound, sound volume,pattern, shape, and time.
 5. The media of claim 1, wherein said computerfurther associates an implication to the conceptual pattern of thearrangement constructed by the respondent based on said comparing step,wherein the implication corresponds to a pattern of the pre-definedrubric and wherein each implication comprises information relating to anoutcome that occurs as the result of a pattern being found in theconceptual pattern of the arrangement constructed by the respondent. 6.A computer-readable storage media having stored thereoncomputer-executable instructions for causing a computer to perform,either alone or in conjunction with one or more users, a method ofcreating a constructed response assessment item, wherein the computerperforms steps comprising: presenting a stimulus response stem requiringa respondent to demonstrate knowledge of a relationship between two ormore concepts; providing at least one device uniquely associated witheach of the two or more concepts, at least one device having at leastone attribute that can be selectively manipulated by the respondent tochange a relationship of that device with respect to at least one otherdevice to construct an arrangement of devices representing arelationship between the concepts with which the devices in thearrangement are associated; and applying a pre-defined scoring rubricfor the constructed response including a conceptual patterncorresponding to a relationship between the two or more concepts tocompare the arrangement constructed by the respondent with theconceptual pattern of the scoring rubric.
 7. The media of claim 6,wherein said rubric includes a conceptual pattern corresponding to acorrect relationship between the two or more concepts.
 8. The media ofclaim 6, wherein the devices comprise virtual items displayed on thecomputer that the respondent can electronically manipulate and/orposition with respect to each other to create a virtual arrangement ofthe devices demonstrating a relationship between the concepts with whichthe devices are associated.
 9. The media of claim 6, wherein theattribute of each device that can be manipulated includes one or more ofsize, length, height, spatial, orientation, color, direction ofmovement, speed of movement, sound, sound volume, pattern, shape, andtime.
 10. A system for administering a constructed response assessmentitem to a respondent, said system comprising: electronic storage media;an output device; a user input device; and a computer operativelyconnected with said storage media, said output device, and said userinput device and configured to: retrieve a stimulus response stem fromthe storage media, wherein the stimulus response stem requires therespondent to demonstrate knowledge of a relationship between two ormore concepts; present the stimulus response stem to a user with saidoutput device; present to the respondent with said output device atleast one device uniquely associated with each of the two or moreconcepts; enable the respondent to use said user input device toselectively manipulate at least one attribute of the at least one deviceto change a relationship of that device with respect to at least oneother device to construct an arrangement of devices representing arelationship between the concepts with which the devices in thearrangement are associated; store in said storage media the attributesof the devices in the arrangement constructed by the respondent; convertthe stored attributes into a conceptual pattern associated with thearrangement constructed by the respondent; retrieve a pre-definedscoring rubric from said storage media, the pre-defined scoring rubricincluding a conceptual pattern corresponding to a relationship betweentwo or more concepts; and compare the conceptual pattern of thearrangement constructed by the respondent with the conceptual pattern ofthe scoring rubric to determine whether or not the conceptual patternassociated with the arrangement constructed by the respondent matchesthe conceptual pattern of the scoring rubric.
 11. The system of claim10, wherein said user input device comprises one or more of a mouse,keyboard, joystick, microphone, remote control unit, camera, tactilesensor, or biometric sensor.
 12. The system of claim 10, wherein saidoutput device comprises one or more of a display, joystick feedbackcomponents, speakers, or actuators.